1.1 Chapter Overview

• Up until now, we have only talked about experimental designs that deal with one manipulated/independent variable and one measured/dependent variable.

• Question: Prior to experiments, we dealt with designs only focusing on measured variables, what were those called?
• A) Correlational
• B) Observational
• C) Probabilistic
• D) Multivariate

• However, we have designs that can look at two (or more) IVs at once and see their individual and combined impact on the DV!

• We refer to these as factorial designs, and they can be helpful in unpacking nuance in certain relationships.

2.1 Overview

• We can add a second (and third) independent variable if we are curious about more than one.

• In addition to the individual effects of both of the IVs, we also get an interaction effect that describes how they change each other’s relationship with the outcome.

• Question: We still have to be concerned about causality, because this is still an experiment - what validity is that most related to?
• A) External
• B) Statistical
• C) Internal
• D) Construct

• Statistically, we might say this interaction is a “difference in differences”
  • Practically, this means that the differences between our groups may be different based on some other trait.
  • More on this later

2.2 Intuitive Interactions

• When confronted with a causal relationship, sometimes we might say, “well it depends” - what it depends on is the second (or third) IV

• Discuss: Previously, we described two types of variables that play a role in the casual link between two others. What were they and describe each? (hint: they both start with the letter M)

• We can see this even in our personal experiences, and many relationships do depend on other factors

• Example: I am assessing how spicy I like my food (on a scale of 1 to 10; my outcome). First, is it cold or hot outside (IV 1)? Second, am I eating Thai or Italian (IV 2)? It is possible that my answer will be different based upon both of the IVs.

• 4 Possible Outcomes:

• I like all of my food spicier when it is hot - Weather effect, but not food

• I like Thai food spicier that Italian, regardless of weather - Food effect, but not temperature

• Whether I like by food spicy or not depends on both the weather, and type of food - interaction effect

• Specifically, we are looking to see whether we have a crossover interaction, like in the graph below:

• My preference for spice doesn’t change, regardless of food type or weather - null findings

2.3 Study Two IVs

• When we work with more than one IV, we use a factorial design.

• This creates more outcome unique conditions = # of Conditions in IV 1 x # of Conditions in IV 2 = total number of conditions

• Both IVs do not have to be manipulated. Often, one will be some categorical, measured trait (e.g., gender, ethnicity, etc.)

  • Sometimes these get called “quasi-manipulated” because they are not actually manipulated but we treat them as such.

• In addition to our statistics, we should show these differences in plots! Interaction effects become especially clear with visual evidence.

2.4 Limit Testing

• Factorial designs can help us find whether outcomes are different for different types of people, or maybe an intervention only works if another intervention is present

• A strong intervention may not be as effective in a different group of people.

• Question: Review - When it comes to a 'strong' intervention what aspect of statistical validity will be mostly looking at?
• A) Precision
• B) Significance
• C) Reproducability
• D) Effect Size

• This can be a boon to our external validity, as we demonstrate findings in a more heterogeneous group.

• We also can establish whether one variable appears to moderate another on the relationship with the outcome variable.

2.5 Test Theories

• For some theoretical reasons, we may have good reason to believe that an effect differs based on some demographic variable.

• Example: I have a new intervention meant to encourage flexibility in learning and taking in new content. However, I recognize that the neuroplasticity of older adults is just lesser in general. Therefore, I believe my intervention will likely be more effective for younger adults, than it will for older adults.

• In essence, we may be able to add nuance and “it depends” to our hypotheses and investigate with factorial designs.

• Discuss: Try coming up with another example of a situation where you might find an effect 'depends' on something else? (Maybe something in your own proposal)

2.6 Main Effects & Interactions

• Main Effects are those that come from each IV on the outcome.
  • The main effect is calculated as an average over the levels of the other IV. Similar to how we “control” for other variable in multiple regression.
  • You have 1 main effect for each IV
• Marginal Means are the statistic that we use to determine whether a main effect is present
  • We can test significance by taking the difference of the two marginal means, and calculating 95% CIs. If CIs contain 0 \(\rightarrow\) non-significant

• An interaction effect can be detected by looking at the differences of the main effect differences. If they are significantly different from one another, then we would say that there is an interaction effect
  • Interactions are often treated as more important, theoretically, that main effects - when they are significant.
• Conventional wisdom: If interaction is significant, focus on that mostly. If interaction is non-significant, focus on main effects of IVs.
  • Interpreting the main effects with a significant interaction can be leaving out important information!

• Discuss: Try explaining, in your own words, why you suspect only looking at the main effects is a bad idea when you have a significant interaction?

• Stats sidebar: This type of analysis is usually done via Two-way ANOVA, which does all the work of calculating significance of interactions, and main effects for us.

3.1 Overview

• Just like with other experiments, we can lay out a factorial design as being between-groups or within-groups.

• But, we can designate each variable as between or within, leading to a total of 3 possible designs:

  • Independent-Groups Factorial
  • Within-Groups Factorial
  • Mixed Factorial

3.2 Independent-Groups Design

• This is when all IVs are between-groups (i.e., participants are arranged into entirely separate groups)

• One nuance is that this will likely require the largest sample size, as each group will have about 1/4th the total number of participants

• Discuss: What validity does a smaller sample size primarily hurt. Explain why this happens

• Example: I measure whether honors or regular students have differential benefits when they are place in living-learning or regular dormitories

3.3 Within-Groups Design

• Much like with previous within-groups designs, this is when participants see every possible condition.

• One thing to watch out for is the need for counterbalancing to prevent order effects

  • Think about how many permutations of condition orders you may need!

• Example: I am interested in seeing whether a certain note-taking strategy and a review strategy help performance on a test - so I have the same people counterbalances to different combinations of both conditions.

3.4 Mixed Factorial Design

• This is when one IV is between-groups, and the other is within-group.

• This is fairly common if we have one demographic variable (between-groups) and one manipulated variable that both demographics are exposed to each level (within-groups).

• Example: I am determining whether first-generation college students or legacy students benefit more from a mentoring program, in terms of confidence. Both first-gen and legacy students have a period that they are exposed to the mentoring program and a time period without.

3.5 More Conditions

• Many IVs are going to naturally have more than one level
  • E.g., race, ethnicity, gender, etc.
• We can use these in factorial designs all the same - and we write it as: \(A x B\) Design.
  • Where \(A\) = Number of conditions in IV 1
  • Where \(B\) = Number of conditions in IV 2
• Statistics here get more complex to interpret - but a good starting point is to try to use a line plot just like what we have done previously and see if lines cross or are parallel.

3.6 More IVs

• Prof. Paul Moes: “God himself cannot interpret a 4-way interaction - neither can you”

• We can do 3 IVs, but with each additional variable the interpretation becomes exponentially more difficult and complicated.

  • One popular alternative is to do this as a multiple regression model instead
  • Stat sidebar: ANOVA and linear regression are both types of the general linear model, so, in a roundabout way, these are actually equivalent!

• Remember to think carefully about what sorts of conclusions you can draw with a design before you use it, and whether an alternative provides a more simpler conclusion.

• Question: Review - we previously emphasized the importance of keeping conclusions as simple as possible, what vocab word captures that sentiment?
• A) Pattern
• B) Purpose
• C) Parsimony
• D) Promise

4.1 Reading Empirical Articles

• Look for words like …

  • “Two-way ANOVA”
  • “Factorial”
  • “Interaction” or “Main Effects”

• You may also see phrasing like “2 x 2 design”, referring to the two conditions of each IV.

• Sometimes a multiple regression model and a factorial design can be described in somewhat similar conclusions - you might have to work through the results to figure out which one was used

4.2 In Popular Media

• Look for words like …
  • “depends”
  • “Only when”
• You may also look for demographic variables …
  • “For males this was the results, but for females…”

4.3 End of Lecture

• Review of Discussion and MC Questions

• Remember to do the Q & A / Lecture Check-in!

Week 12 Lecture - Complex Experiments || Undergraduate Research Methods in Psychology